Methods and devices for grading a medical image

ABSTRACT

Method and system for grading a medical image. For example, a system for grading a medical image comprising a grading network configured to provide a grading result corresponding to the medical image based on at least the medical image and/or a list of lesion candidates generated by a lesion identification network.

1. CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to Chinese Patent Application No.201811610713.4, filed Dec. 27, 2018, incorporated by reference hereinfor all purposes.

2. BACKGROUND OF THE INVENTION

Certain embodiments of the present invention are directed to imageprocessing. More particularly, some embodiments of the invention providesystems and methods for grading a medical image. Merely by way ofexample, some embodiments of the invention have been applied todiagnosing a medical image. But it would be recognized that theinvention has a much broader range of applicability.

X-ray chest radiographs (or chest radiographs obtained in other methods)are commonly used in clinical examination. X-ray examinations (e.g.,which generate chest radiographs) are often performed for generaloutpatient, emergency, and intensive care unit (ICU) to help diagnoseand observe a patient's health. In physical examinations (e.g., as partof general health examinations or workforce entrance examinations),X-ray chest radiographs are also one of the standard examination itemsfor checking whether there are any abnormal conditions in the chest of asubject (e.g., the patient). Therefore, the amount of X-ray chestradiographs produced each year is very large. As a result, the workloadof reading X-ray chest radiographs in the radiology department is verylarge. In a typical physical examination session, most of the X-raychest radiographs to-be-examined are normal, with the abnormal X-raychest radiographs accounting for only a small portion of the totalamount of the X-ray chest radiographs. As a result, the work of X-rayradiography diagnosis is relatively uneconomical in respect to theoverall work of the radiology department. There is a need to improve theefficiency of clinical reading.

Triage is a method, process, or procedure for determining the priorityof patients' treatments based on the severity and/or urgency of theircondition. In some examples, the implementation of triage includesprompting a specialist, a doctor, a medical staff, or a medical team totake different measures according to the severity and/or urgency. Forexample, priority is given to treating the most critical patients,thereby maximizing the application efficiency of limited medicalresources. The triage can be implemented in the emergency department,the treatment of wounded soldiers, and the telephonic consultations forpre-screening conditions. As the use of triage grow in popularity anddemand in the radiology departments, such as for medical imaging anddiagnosis, implementing triage in medical imaging applications hasbecome increasingly important.

At present, conventional X-ray chest radiograph clinical workflow doesnot have prioritization functions (e.g., via an intelligent software),instead, the order of diagnostic analysis (or interpretation) is mostlyperformed in the order of first-in first-read. As such, concerns withthe workflow is understandable due to the uneconomical and inefficientnature. For example, such concerning workflow is implemented in thephysical examination applications described above. In the emergencyapplication scenarios, although there are currently regulationsrequiring radiologists to diagnose and report on emergency medicalimages within 30 minutes. However, for some particularly urgentsituations, such as during a shortage of radiologists in the middle ofthe night, the efficiency of the entire workflow has room forimprovement. A specific application of the X chest radiograph severityclassification method in an emergency scenario is, for example, theability to prioritize cases of special emergency to the emergencydepartment doctor or the corresponding doctor (e.g., the doctoron-staff), to allow the radiologist to quickly judge and reduce thewaiting time (e.g., of up to 30 minutes). In the application scenario ofthe ICU, bedside chest X-rays are often used daily to observe the healthstatus of critically ill patients, who may be in stable condition or innotable conditions. In emergency applications such as in the intensivecare unit, grading (or classifying) of X-ray chest radiographs can helpin identifying lung conditions, heart conditions, cardiovascular status,hilar conditions, bone tissue status, and/or pleural conditions. Forexample, a radiology doctor can first read and diagnose the urgentradiographs, and timely implement appropriate treatment improvements tohelp the patient.

In some scenarios, the intelligent application of classification ofX-ray chest radiographs in the emergency clinical medical imagediagnosis process, the applicable scenarios are very extensive, withclear need in the medical field for improved clinical work efficiency.

Conventional methods, however, are absent of such intelligentapplication products or solutions for X-ray chest radiographs diagnosis.

Therefore, there is a need for applying triage intelligent applicationfor a wide variety of X-ray chest radiograph diagnosis procedures foridentifying a wide range of chest conditions (e.g., lung condition,heart condition, cardiovascular condition, hilar condition, bone tissuecondition, pleural condition, etc.) to improve the efficiency ofworkflows such as physical examination workflows in emergency servicesand intensive care units.

Technically, however, applying artificial intelligence technology tograding the severity and/or urgency of an image content of an X-raychest radiograph is difficult. This may be due to the degree of urgencybeing closely related to the type of disease and the effects of abnormalphenomena, and the need to consider multiple diseases and multipleorgans in the chest. As such, it is more complicated and difficult thanapplying artificial intelligence technology to diagnosing a singledisease (e.g., lung nodules, breast lesions, or strokes). Specifically,for example, it is necessary to consider a plurality of diseases in thechest organ such as the lung and/or the heart, such as pulmonaryeffusion, emphysema, pneumothorax, cardiac hypertrophy, or the like, ormultiple abnormal symptoms such as mediastinal thickening, aortic archtortuosity, etc. Various other lung conditions, cardiac conditions,cardiovascular conditions, hilar conditions, bone tissue status, pleuralconditions, etc., need to be considered to effectively implement themethod of chest urgency emergency grading. Known conventional methodsare not yet able to grade X-ray chest radiographs by severity and/orurgency.

3. BRIEF SUMMARY OF THE INVENTION

Certain embodiments of the present invention are directed to imageprocessing. More particularly, some embodiments of the invention providesystems and methods for grading a medical image. Merely by way ofexample, some embodiments of the invention have been applied todiagnosing a medical image. But it would be recognized that theinvention has a much broader range of applicability.

In various embodiments, a system for grading a medical image includes agrading network configured to provide a grading result corresponding tothe medical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network.

In some embodiments, the grading network is further configured toprovide the grading result corresponding to the medical image based onat least the medical image and the list of lesion candidates generatedby the lesion identification network.

In some embodiments, the grading network is further configured toprovide the grading result corresponding to the medical image based onat least the medical image.

In some embodiments, the lesion identification network is configured toprovide a score corresponding to the list of lesion candidates. In someexamples, the grading result is provided further based on the score. Incertain examples, the score is provided based on at least a probabilityvalue corresponding to the medical image including lesions of multiplelesion categories.

In some embodiments, the medical image includes a chest radiograph. Insome examples, the list of lesion candidates includes at least one listselected from a group consisting of a list of lung lesion candidates, alist of cardiac lesion candidates, a list of cardiovascular lesioncandidates, a list of hilar lesion candidates, a list of bone tissuelesion candidates, and a list of pleural lesion candidates.

In some embodiments, the grading network is further configured toprovide one or more outputs to guide manual scheduling of a plurality ofradiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs. In some examples, the grading network is further configuredto automatically schedule the plurality of radiographs for manual reviewand diagnosis based on at least a plurality of grading resultscorresponding to the plurality of radiographs.

In some embodiments, the grading network is further configured toprovide one or more outputs to guide manual scheduling of a plurality ofradiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs, and automatically schedule the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.

In some embodiments, the grading network is a student network trained byan attention transfer learning process includes: establishing a teachernetwork and the student network for the grading network; training theteacher network; and training the student network based on at least:extracting a feature map from one or more middle layers corresponding toboth the student network and the teacher network; calculating one ormore attention transfer learning losses corresponding to the one or moremiddle layers; and backpropagating the one or more attention transferlearning losses into the student network.

In some embodiments, the grading result corresponds to at least oneselected from a group consisting of severity and urgency.

In some embodiments, the lesion identification network is configured toidentify at least one lesion characteristic selected from a groupconsisting of color, shape, size, grayscale value, position, andmorphology.

In some embodiments, the grading result is provided based on the medicalimage as a whole or one or more partial regions of the medical image.

In some embodiments, the grading result is selected from a groupconsisting of a first grade, a second grade, and a third grade, wherein:the first grade corresponds to a first priority for reading anddiagnosis; the second grade corresponds to a second priority for readingand diagnosis; the third grade corresponds to a third priority forreading and diagnosis; the first priority being greater than the secondand third priorities.

In various embodiments, a computer-implemented method for grading amedical image includes providing a grading result corresponding to themedical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network.

In some embodiments, the computer-implemented method further includesproviding the grading result corresponding to the medical image based onat least the medical image and the list of lesion candidates generatedby the lesion identification network.

In some embodiments, the computer-implemented method further includesproviding the grading result corresponding to the medical image based onat least the medical image.

In some embodiments, the computer-implemented method further includesproviding a score corresponding to the list of lesion candidates usingthe lesion identification network. In some examples, the providing agrading result is further based on the score. In certain examples, theproviding a score is based on at least a probability value correspondingto the medical image including lesions of multiple lesion categories.

In some embodiments, the computer-implemented method further includesproviding one or more outputs to guide manual scheduling of a pluralityof radiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs. In some examples, the computer-implemented method furtherincludes automatically scheduling the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.

In some embodiments, the computer-implemented method further includestraining a student network to be the grading network based on anattention transfer learning process includes: establishing a teachernetwork and the student network for the grading network; training theteacher network; and training the student network based on at least:extracting a feature map from one or more middle layers corresponding toboth the student network and the teacher network; calculating one ormore attention transfer learning losses corresponding to the one or moremiddle layers; and backpropagating the one or more attention transferlearning losses into the student network.

In some embodiments, scheduling of the plurality of radiographs formanual review and diagnosis is based on at least one selected from agroup consisting of expertise, pay level, and seniority.

In various embodiments, a non-transitory computer-readable medium withinstructions stored thereon, that when executed by a processor, performthe process of: providing a grading result corresponding to the medicalimage based on at least the medical image and/or a list of lesioncandidates generated by a lesion identification network.

4. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram showing a deep-learning neural networkfor grading an X-ray chest radiograph by severity and/or urgency,according to some embodiments of the present invention.

FIG. 2 is a simplified diagram showing a system for training adeep-learning neural network for grading an X-ray chest radiograph byseverity and/or urgency, according to some embodiments of the presentinvention.

FIG. 3A is a simplified diagram showing a method for grading an X-raychest radiograph by severity and/or urgency, according to someembodiments of the present invention.

FIG. 3B is a simplified diagram showing a method for grading an X-raychest radiograph by severity and/or urgency using artificialintelligence, according to some embodiments of the present invention.

FIG. 4 is a simplified diagram showing a method for grading an X-raychest radiograph by severity and/or urgency in an emergency or intensivecare unit scenario, according to some embodiments of the presentinvention.

FIG. 5 is a simplified diagram showing a method for grading an X-raychest radiograph by severity and/or urgency in a physical examinationscenario, according to some embodiments of the present invention.

FIG. 6 is a simplified diagram showing a system for grading an X-raychest radiograph by severity and/or urgency, according to someembodiments of the present invention.

FIG. 7 is an illustrative representation of an interface of a system forgrading an X-ray chest radiograph by severity and/or urgency, accordingto some embodiments of the present invention.

5. DETAILED DESCRIPTION OF THE INVENTION

Certain embodiments of the present invention are directed to imageprocessing. More particularly, some embodiments of the invention providesystems and methods for grading a medical image. Merely by way ofexample, some embodiments of the invention have been applied todiagnosing a medical image. But it would be recognized that theinvention has a much broader range of applicability.

In some examples, the present disclosure relates to medical imageassessment, and more particularly to the use of neural networks forsmart medical image assessment.

In some examples, the present disclosure relates to a system for gradinga medical image (e.g., by severity) including a grading (e.g.,hierarchical) network configured to provide a grading (e.g., ranking)result at least based on an input medical image (e.g., that has beeninput into the system) and/or a list (e.g., listing) of lesioncandidates generated by a lesion identification network. In certainembodiments, the present disclosure also relates to corresponding methodand non-transitory computer-readable medium.

A grading network may be referred to as a hierarchical network. A neuralnetwork may be referred to as a network. A lesion identifying neuralnetwork may be referred to as a lesion recognizing neural network. Achest radiograph may be referred to as a film. An X-ray chest radiographmay be referred to as an X chest radiograph. Transfer learning may bereferred to as migration learning. An identifying network may bereferred to as an identification network. X-ray chest radiographs may bereplaced by similar medical images, such as ones obtained by methodsother than X-ray examination. Reading and diagnosis of a chestradiograph may be referred to as manual review and diagnosis of a chestradiograph.

In various embodiments, a system for grading a medical image includes agrading network configured to provide a grading result corresponding tothe medical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network.

In various examples, the lesion identification network is configured toprovide a score corresponding to the list of lesion candidates. In someexamples, the grading result is provided further based on the score. Insome examples, the score is provided based on at least a probabilityvalue corresponding to the medical image including lesions of multiplelesion categories.

In various examples, the medical image includes a chest radiograph. Insome examples, the list of lesion candidates includes one or more listsselected from a group consisting of a list of lung lesion candidates, alist of cardiac lesion candidates, a list of cardiovascular lesioncandidates, a list of hilar lesion candidates, a list of bone tissuelesion candidates, and a list of pleural lesion candidates.

In various examples, the system for grading a medical image isconfigured to provide one or more prompts to guide manual scheduling ofa plurality of radiographs for manual review and diagnosis based on atleast a plurality of grading results corresponding to the plurality ofradiographs. In some examples, the system for grading a medical image isconfigured to automatically schedule the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.

In various examples, the grading network is trained by an attentiontransfer learning method. In some examples, training the grading networkby the attention transfer learning method includes establishing ateacher network and a student network for the grading network, trainingthe teacher network; and training the student network based on at leastextracting a feature map from one or more middle layers corresponding toboth the student network and the teacher network calculating one or moreattention transfer learning losses corresponding to the one or moremiddle layers; and backpropagating the one or more attention transferlearning losses into the student network.

In various embodiments, a computer-implemented method for grading amedical image includes providing a grading result corresponding to themedical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network.

In various examples, the computer-implemented method further includesproviding a score corresponding to the list of lesion candidates usingthe lesion identification network. In some examples, the providing agrading result is further based on the score. In some examples, theproviding a score is based on at least a probability value correspondingto the medical image including lesions of multiple lesion categories.

In various examples, the computer-implemented method further includesproviding one or more prompts to guide manual scheduling of a pluralityof radiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs. In some examples, the computer-implemented method furtherincludes automatically scheduling the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.

In various examples, the grading network is trained by an attentiontransfer learning method. In some examples, training the grading networkby the attention transfer learning method includes establishing ateacher network and a student network for the grading network, trainingthe teacher network, and training the student network based on at leastextracting a feature map from one or more middle layers corresponding toboth the student network and the teacher network, calculating one ormore attention transfer learning losses corresponding to the one or moremiddle layers, and backpropagating the one or more attention transferlearning losses into the student network.

In various embodiments, a non-transitory computer-readable medium withinstructions stored thereon, that when executed by a processor, performthe process of providing a grading result corresponding to the medicalimage based on at least the medical image and/or a list of lesioncandidates generated by a lesion identification network.

In various embodiments, the disclosure relates to a system for grading amedical image including a grading network configured to provide one ormore grading results based on at least one or more input medical imagesand/or a list of lesion candidates generated by a lesion identificationnetwork. The lesion identification network is part of the gradingnetwork or is a separate network from the grading network.

In various embodiments, the present disclosure relates to an artificialintelligence application for grading of one or more X-ray chestradiographs based on severity and/or urgency. In certain examples, thepresent disclosure systems and methods for identifying conditions,lesions, diseases, and the like in X-ray chest radiographs by using deeplearning techniques, such as by providing grades and/or scores based onseverity and/or urgency. In some examples, by using a deep learningneural network, the severity of an X-ray chest radiograph (which may bereferred to as a film) can be graded, such as in a use case of aradiology department workflow. For example, the most severe chestradiographs can be placed at the top of a work list of the radiologydepartment, followed by the less serious chest radiographs, and thelikely-normal chest radiographs are placed later in the work list. Incertain examples, the diagnosis of chest radiographs that are identifiedas likely-normal can be completed by appropriate division of labor andtime, thereby improving the working efficiency of the clinicalradiograph-analysis.

It should be understood that the described embodiments are only a partof the embodiments of the invention, and not all of the embodiments. Allother embodiments obtained by those skilled in the art based on theembodiments of the present invention without creative efforts are withinthe scope of the present invention.

FIG. 1 is a simplified diagram showing a deep-learning neural network100 for grading an X-ray chest radiograph by severity and/or urgency,according to some embodiments of the present invention. This diagram ismerely an example, which should not unduly limit the scope of theclaims. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. In some examples, theneural network 100 includes a plurality of neural network layersincluding L amount of neural network layers and a fully-connected layer.In certain examples, the plurality of neural network layers includes aneural network layer 1 (first neural network layer), a neural networklayer 2 (second neural network layer), and a neural network layer L(L^(th) neural network layer). Although the above has been shown using aselected group of components for the neural network, there can be manyalternatives, modifications, and variations. For example, some of thecomponents may be expanded and/or combined. Other components may beinserted to those noted above. Depending upon the embodiment, thearrangement of components may be interchanged with others replaced.

In some embodiments, the neural network layer 1 includes a convolutionallayer, a batch normalization (BN) layer, an activation layer, andoptionally a pooling layer. In certain examples, one or more of theneural network layer 2 . . . the neural network layer L and the likealso includes one or more of the convolutional layer, batchnormalization (BN) layer, activation layer, and pooling layer describedabove. In certain embodiments, the neural network layer 1 is configuredto receive an input image, which may be labeled as F₀. In some examples,the convolutional layer in the neural network layer 1 is configured toperform feature extraction on the input image F₀. In certain examples,the BN layer in the neural network layer 1 is configured to receive thefeature(s) extracted by the convolutional layer and to normalize theoutput from the convolutional layer. In various examples, the activationlayer in the neural network layer 1 is configured to receive the output(e.g., normalized) of the BN layer and apply an activation function toit to incorporate one or more nonlinear factors. In some embodiments,the pooling layer in the neural network layer 1 is configured to receivethe output of the activation layer and to compress the feature(s). Invarious embodiments, the system 100 is configured to output the resultafter pooling or prior pooling of the neural network layer 1 as a firstfeature map (labeled as F₁). In some examples, the system 100 isconfigured to input the first feature map F₁ into the neural networklayer 2. In certain embodiments, the convolutional layer, the BN layer,the activation layer, and/or the pooling layer in the neural networklayer 2 is configured to process the first feature map F₁ and output asecond feature map (labeled as F₂), wherein such process is repeatedsimilarly for the remaining neural network layers in the plurality ofneural network layers. As illustrated, the neural network layer L isconfigured to output a L^(th) feature map (labeled F_(L)) to the fullyconnected layer. In some examples, the fully connected layer isconfigured to connect all the features and to output a probability value(labeled P). In various embodiments, the system 100 further includes aclassifier (e.g., a soft-max layer) (not shown) is configured to receivethe probability value P. In certain examples, the soft-max layer isconfigured as an output layer, such as a last layer of the neuralnetwork system 100, wherein the system 100 may be used formulti-classification (e.g., of lesion types in one or more organs). Invarious embodiments, the classification includes a single lesionclassification and/or multiple lesion classifications. In some examples,more or less levels of neural network may be present in the neuralnetwork system 100. In various examples, the neural network isconfigured to be emulated via software by a general-purpose orspecific-purpose hardware (e.g., circuitry).

In some embodiments, the neural network of the present disclosure forgrading an X-ray chest radiograph by severity and/or urgency can betrained using deep learning methods, such as trained using multipleX-ray chest radiographs. In various embodiments, training the gradingnetwork 100 and/or a lesion identification network (e.g., as part of orseparate from the grading network) is performed in a fully supervised orweakly supervised manner (e.g., selected based on the degree ofcompleteness of data-labeling). For example, when the grading network istrained with training samples absence of sufficient annotated data, thetraining is conducted in a weakly supervised manner. In another example,when the grading network is trained with training samples withsufficiently (e.g., large amount or fully) annotated training samples(e.g., with clear and/or clean data), the training is conducted in afully supervised manner.

In certain embodiments, the grading network 100 is configured to betrained using an attention transfer training method to improve thetraining of the deep convolutional neural network (e.g., the gradingnetwork 100), as there are increasing amount of new data and/or newlesion categories made available during training of the grading network100. In certain examples, in the training of the deep convolutionalneural network (e.g., the grading network 100), attention includesfeature-based (e.g., color, shape, etc.) attention (FBA), spatial-based(e.g., position) attention, and/or the like. In some embodiments, usingan attention transfer training method includes learning a source domainto solve a target domain. For example, classification of known lesionscan be learned or transferred during training to enable classificationof new lesions.

FIG. 2 is a simplified diagram showing a system 200 for training adeep-learning neural network for grading an X-ray chest radiograph byseverity and/or urgency, according to some embodiments of the presentinvention. This diagram is merely an example, which should not undulylimit the scope of the claims. One of ordinary skill in the art wouldrecognize many variations, alternatives, and modifications. In someexamples, the system 200 includes a teacher network (a first network)and a student network (a second network). Although the above has beenshown using a selected group of components for the system, there can bemany alternatives, modifications, and variations. For example, some ofthe components may be expanded and/or combined. Other components may beinserted to those noted above. Depending upon the embodiment, thearrangement of components may be interchanged with others replaced.

In various embodiments, the teacher network and/or the student networkof the training system 200 includes a neural network structure similarto the structure of neural network 100 of FIG. 1. In some examples, theteacher network is a classification network for known lesions, and thestudent network is a classification network for new lesions. In certainembodiments, the teacher network is first trained. In various examples,the student network is next trained, wherein one or more feature maps ofone or more middle (or intermediate) layers (e.g., neural networklayers) corresponding to both the teacher network and the studentnetwork are extracted for calculating one or more corresponding transfer(migration) learning losses, and wherein the one or more transferlearning losses is back-propagated into the student network. In variousexamples, the system 200 is configured to calculate transfer learningloss l₁ based on the feature learning map F₂ from the feature map fromthe neural network layer 2 of the teacher network and input the transferlearning loss l₁ into the neural network layer 3 of the student network.Such calculation and input of transfer learning loss is repeated for themultiple neural network layers of the system 200. In certainembodiments, the transfer learning loss l_(m) is calculated from thefeature map F_(L) output from the neural network layer L of the teachernetwork, and input into the fully connected layer of the studentnetwork. Finally, the trained student network is used for classifying amedical image, such as used for generating a grade based on severityand/or urgency. For example, for an input image F₀, the student networkis configured to output a probability value P_(s). In suchteacher-student learning method, the teacher network's learned knowledgein the classification of known lesions can be transferred to the studentnetwork's ability classify new lesions.

FIG. 3A is a simplified diagram showing a method 300 for grading anX-ray chest radiograph by severity and/or urgency, according to someembodiments of the present invention. This diagram is merely an example,which should not unduly limit the scope of the claims. One of ordinaryskill in the art would recognize many variations, alternatives, andmodifications. In some examples, the method 300 includes a process 301of receiving a chest radiograph, a process 302 of identifying a disease(one or more) and/or an (one or more) abnormal lesion based on at leastthe chest radiograph, a process 303 of integrating the identificationresults, and a process 304 of providing a score (e.g., based on severityand/or urgency) corresponding to the chest radiograph based on at leastthe identification results. Although the above has been shown using aselected group of processes for the method, there can be manyalternatives, modifications, and variations. For example, some of theprocesses may be expanded and/or combined. Other processes may beinserted to those noted above. Depending upon the embodiment, thesequence of processes may be interchanged with others replaced.

FIG. 3B is a simplified diagram showing a method 300′ for grading anX-ray chest radiograph by severity and/or urgency using artificialintelligence, according to some embodiments of the present invention.This diagram is merely an example, which should not unduly limit thescope of the claims. One of ordinary skill in the art would recognizemany variations, alternatives, and modifications. In some examples, themethod 300′ includes an optional first leg of processes (a) including aprocess 311 of inputting a chest radiograph into a trained lesionidentifying neural network, and a process 312 of identifying a list oflesion candidates and/or assigning a score associated with the chestradiograph, and a second leg of processes (b) including a process 314 ofusing the identified list of lesion candidates and/or the scoreassociated with the chest radiograph, a process 315 of inputting thechest radiograph, the identified list of lesion candidates, and/or thescore associated with the chest radiograph into a grading network, and aprocess 316 of assigning a grade (e.g., a label) associated with thechest radiograph. Although the above has been shown using a selectedgroup of processes for the method, there can be many alternatives,modifications, and variations. For example, some of the processes may beexpanded and/or combined. Other processes may be inserted to those notedabove. Depending upon the embodiment, the sequence of processes may beinterchanged with others replaced.

In various embodiments, implementation of the grading of an X-ray chestradiograph based on urgency and/or severity includes using a lesionidentifying neural network and/or a grading (e.g., hierarchical)network. In some examples, the trained lesion identifying neural networkis configured to identify one or more lesion candidates based on atleast the chest radiograph. In certain examples, the trained lesionidentifying neural network is configured to calculate, determine, and/orassign a score corresponding to the identified one or more lesioncandidates, which can be represented as a list of lesion candidates. Invarious examples, the trained lesion identifying neural network isconfigured to extract one or more features or characteristics of eachlesion candidate, such as color, shape, size, grayscale value, position,and/or morphology. In some embodiments, the trained lesion identifyingneural network is deep learning-based. In certain embodiments, thetrained lesion identifying neural network is configured to identify (orrecognize), such as automatically, multiple diseases (or candidates ofdiseases) and/or multiple (abnormal) lesions (or candidates of lesions),such as concurrently. In some examples, assigning the scorecorresponding to the identified one or more lesion candidates includesassigning one or more scores each corresponding to one of the identifiedone or more lesion candidates. In certain embodiments, the lesionidentifying neural network includes a soft-max layer configured togenerate the score (assessment score) corresponding to a chestradiograph. For example, the soft-max layer is configured to output aN-dimensional vector, wherein a value of the n^(th) dimension in thevector is the probability value that the chest radiograph belongs to an^(th) category (or classification) of a disease or lesion. In someexamples, a process of providing a score (e.g., the process 304)corresponding to a chest radiograph includes providing a probability orprobability value as the score (e.g., score equals to the probability).In certain examples, each dimension of the N-dimensions of theN-dimensional vector corresponds to a category of disease or lesion. Invarious embodiments, the soft-max layer is configured to map elements ofthe N-dimension vector v_(i) to (0, 1), sum the mapped elements to asummed value of 1 to satisfy the probability property, and maintain theoriginal order of the elements based on their original element sizes. Insome embodiments, the soft-max layer is configured to output aclassification target with a highest probability value corresponding toa mapped element, which the lesion identifying neural network, in someexamples, is configured to consider the classification target as aprimary lesion candidate among the identified one or more lesioncandidates. In certain embodiments, the lesion recognition neuralnetwork is configured to automatically identify multiple lesions (e.g.,different) on a medical image, with or without a feature extractionmodule configured to extract one or more features of a lesion.

In various embodiments, the trained lesion identification network isconfigured to output the list of lesion candidates and theircorresponding scores for use as analysis results that are to be inputinto the grading network. In certain examples, the trained lesionidentification network is configured to provide a score corresponding tothe list of lesion candidates. In some examples, the score or gradeassigned by the trained identification network and/or by the gradingnetwork is based on the type, the severity, the urgency, and/or theprobability (or confidence level) of the lesion identified (e.g., by thetrained lesion identification network). In various examples, theprobability value indicates the probability of a lesion to be one of thelesion candidates included in the list of lesion candidates.

In some examples, the process 315 of inputting the chest radiograph, theidentified list of lesion candidates, and/or the score(s) associatedwith the chest radiograph and/or the list of lesion candidates into thegrading network is performed without inputting one or more of the chestradiograph, the identified list of lesion candidates, and the score(s)associated with the chest radiograph. In various examples, the trainednetwork (e.g., including the trained lesion identifying neural network)is configured to grade, rank, label, and/or sort one or more chestradiographs, such as based on at least the analysis results of thelesion identifying neural network and/or the chest radiographs (as inthe process 316). In an example, the trained network (e.g., includingthe trained lesion identifying neural network) is configured to grade,rank, label, and/or sort one or more chest radiographs based on at leastthe analysis results (e.g., a list of lesion candidates) of the lesionidentifying neural network. In another example, the trained network(e.g., including the trained lesion identifying neural network) isconfigured to grade, rank, label, and/or sort one or more chestradiographs based on at least the chest radiographs. In certainexamples, the grading network is configured to generate a grading result(e.g., a grade, a rank, a label) corresponding to the criticality,severity, and/or urgency of the chest radiograph as a whole or of one ormore partial regions of the chest radiograph.

In some examples, the lesion identifying neural network and/or thegrading network of FIG. 3A and/or FIG. 3B is the neural network 100depicted in FIG. 1, and in certain examples, is trained by the system200 depicted in FIG. 2. It is to be understood that other neuralnetworks and/or systems and methods for training a neural network canalso be used in various embodiments of the present disclosure.

In certain examples, the training (or learning) of the lesionidentifying neural network and/or the grading network can be performedusing cascaded learning method. In some embodiments using such cascadedlearning method, the process of detecting image features (e.g.,explicitly) is removed to achieve image-level severity and gradualgrading. In some examples, the lesion identification network and thegrading network can be implemented as a single (e.g., merged) network.For example, the grading network includes the lesion identificationnetwork.

In various embodiments, the intelligent grading of X-ray chestradiographs is applicable to in emergency scenarios such as generaloutpatient, physical examination, emergency department, and/or intensivecare unit. For example, some hospitals implement triage when theiremergency resources are limited to maximize the use of medicalresources. In yet another example, medical imaging is a frequently usedinspection tool in emergency situations and can benefit from increase inefficiency of radiograph-reading workflow, such as via using saidintelligent grading of X-ray chest radiographs. FIG. 4, which will bedescribed in more detail, illustrates a method 400 for grading an X-raychest radiograph in an emergency scenario, in accordance with variousembodiments. For example, the method 400 is applicable in intensive careunits, such as ones often use bedside X-ray examinations to observe thehealth status of the patient's chest. In some examples, the method 400includes determining patients who are unstable and patients who exhibitlittle status change (e.g., stable). Such application of method 400 can,especially in emergency scenarios, prevent delay treatment such asbeyond golden treatment time windows. The application of method 400 canimprove treatment results, such as by help prioritizing tasks based onseverity and/or urgency graded by an intelligent network (e.g., network100).

FIG. 4 is a simplified diagram showing a method 400 for grading an X-raychest radiograph by severity and/or urgency in an emergency or intensivecare unit scenario, according to some embodiments of the presentinvention. This diagram is merely an example, which should not undulylimit the scope of the claims. One of ordinary skill in the art wouldrecognize many variations, alternatives, and modifications. In someexamples, the method 400 includes a process 401 of receiving a chestradiograph, a process 402 of inputting the chest radiograph into atrained grading neural network, a process 403 of assigning a grade of“critical” followed by a process 404 of prompting for immediate readingand diagnosis, or a process 405 of assigning a grade of “requireattention” or a process 406 of assigning a grade of “no notableabnormality” followed by a process 407 of prompting for general readingand diagnosis and a process 408 of providing the chest radiograph forreading and diagnosis when selected. In various examples, the method 400is performed by the system 100, such as one trained by system 200, suchas one including the lesion identification network described in method300. Although the above has been shown using a selected group ofprocesses for the method, there can be many alternatives, modifications,and variations. For example, some of the processes may be expandedand/or combined. Other processes may be inserted to those noted above.Depending upon the embodiment, the sequence of processes may beinterchanged with others replaced.

In some examples, the method 400 includes identifying multiple diseasesand/or multiple abnormal lesions, outputting a list of lesion candidatesand/or one or more corresponding scores. For example, the list of lesioncandidates includes a list of lung lesions, a list of heart lesions, alist of cardiovascular lesions, a list of hilar lesions, a list of bonetissue lesions, and/or a list of pleural lesions. In some embodiments,the trained grading neural network (may be referred to as the gradingnetwork) is configured to rank, rate, grade, and/or assign a grade toone or more input X-ray chest radiographs, such as based on at least theanalysis results of the lesion identification neural network and/or theinput X-ray chest radiographs. In some examples, the grading resultsgenerated by the grading network include or are based on, for example,the criticality, severity, and/or urgency of the X-ray chest radiographsin whole or partial regions thereof.

In some embodiments, the grading network is configured to assign one ofa plurality of grades or labels. For example, the plurality of gradesincludes critical (or urgent) (e.g., as assigned in process 403),require attention (or attention-needed) (e.g., as assigned in process405), and no notable abnormality (or no major anomalies) (e.g., asassigned in process 406). In various examples, when the grading resultis that the condition of the patient corresponding to the chestradiograph is “critical” (e.g., as assigned in process 403), the method400 includes performing the process 404 of prompting for immediatereading and diagnosis, such as by a specialist or doctor on staff. Insome examples, the method 400 further includes transmitting data to auser (e.g., administrator, expert, etc.). In some examples, when thegrading result is that the condition of the patient corresponding to thechest radiograph is “require attention” (e.g., as assigned in process405) or “no notable abnormality” (e.g., as assigned in process 406), themethod 400 includes performing the process 407 of prompting for generalreading and diagnosis (e.g., by a radiologist in some later time), andoptionally the process 408 of providing the chest radiograph for readingand diagnosis when selected. Similarly, data can be transmitted to auser.

FIG. 5 is a simplified diagram showing a method 500 for grading an X-raychest radiograph by severity and/or urgency in a physical examinationscenario, according to some embodiments of the present invention. Thisdiagram is merely an example, which should not unduly limit the scope ofthe claims. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. In some examples, themethod 500 includes a process 501 of receiving a chest radiograph, aprocess 502 of inputting the chest radiograph into a trained gradingneural network, a process 503 of assigning a grade of “more urgent”followed by a process 504 of prompting a first priority for reading anddiagnosis, or a process 505 of assigning a grade of “require attention”followed by a process 506 of prompting a second priority for reading anddiagnosis, or a process 507 of assigning a grade of “no notableabnormality” followed by a process 508 of prompting a third priority forreading and diagnosis. In various examples, the method 500 is performedby the system 100, such as one trained by system 200, such as oneincluding the lesion identification network described in method 300.Although the above has been shown using a selected group of processesfor the method, there can be many alternatives, modifications, andvariations. For example, some of the processes may be expanded and/orcombined. Other processes may be inserted to those noted above.Depending upon the embodiment, the sequence of processes may beinterchanged with others replaced.

In various embodiments, the method 500 is applicable in scenarios suchas physical examinations and in general clinics, which in some examples,use X-ray examinations.

In some examples, the method 500 includes identifying multiple diseasesand/or multiple abnormal lesions, outputting a list of lesion candidatesand/or one or more corresponding scores. For example, the list of lesioncandidates includes a list of lung lesions, a list of heart lesions, alist of cardiovascular lesions, a list of hilar lesions, a list of bonetissue lesions, and/or a list of pleural lesions. In some embodiments,the trained grading neural network (may be referred to as the gradingnetwork) is configured to rank, rate, grade, and/or assign a grade toone or more input X-ray chest radiographs, such as based on at least theanalysis results of the lesion identification neural network and/or theinput X-ray chest radiographs. In some examples, the grading resultsgenerated by the grading network include or are based on, for example,the criticality, severity, and/or urgency of the X-ray chest radiographsin whole or partial regions thereof.

In some embodiments, the grading network is configured to assign one ofa plurality of grades or labels. For example, the plurality of gradesincludes critical (or urgent) (e.g., as assigned in process 503),require attention (or attention-needed) (e.g., as assigned in process505), and no notable abnormality (or no major anomalies) (e.g., asassigned in process 507). In various examples, when the grading resultis that the condition of the patient corresponding to the chestradiograph is “critical” (e.g., as assigned in process 503), the method500 includes performing the process 504 of prompting a first priorityfor reading and diagnosis, such as by a specialist or doctor on staff.In some examples, when the grading result is that the condition of thepatient corresponding to the chest radiograph is “require attention”(e.g., as assigned in process 505), the method 500 includes performingthe process 506 of prompting a second priority for reading anddiagnosis, such as by the specialist or doctor on staff after reviewingthe chest radiographs prompted as having first priority. In certainexamples, when the grading result is that the condition of the patientcorresponding to the chest radiograph is “no notable abnormality” (e.g.,as assigned in process 507), the method 500 includes performing theprocess 508 of prompting a third priority for reading and diagnosis,such as by the specialist or doctor after reviewing the chestradiographs prompted as having first priority and chest radiographshaving second priority. In certain embodiments, the process 508 includesprompting for labor and/or time division for reading and diagnosis ofchest radiographs.

In certain examples, the method 400 and/or method 500 includes updatingthe urgency of the radiology department work list and/or reading anddiagnosis priority, such as in real time and/or based on specificsituational requirements of the scenario. For example, a situationalrequirement for some emergency scenarios is for emergency diagnosis andreporting to be completed within thirty minutes of first viewing of thex-ray chest radiograph. In various examples, prioritization of the chestradiographs based on urgency helps automatic division of labor, which insome examples, is based on at least expertise, pay level, and/orseniority (e.g., of each doctor and/or specialist). In some examples,the grading method is customizable appropriately for differentapplications to support suitable diagnostic workflows.

FIG. 6 is a simplified diagram showing a system 600 for grading an X-raychest radiograph by severity and/or urgency, according to someembodiments of the present invention. This diagram is merely an example,which should not unduly limit the scope of the claims. One of ordinaryskill in the art would recognize many variations, alternatives, andmodifications. In some examples, the system 600 is configured to performthe method 100, the method 300, the method 400, and/or the method 500.In some examples, the system 600 includes a processor 601, a memory 602,a lesion identification module 603, a grading module 604, and aninput/output (I/O) interface 605. Although the above has been shownusing a selected group of components for the system, there can be manyalternatives, modifications, and variations. For example, some of thecomponents may be expanded and/or combined. Other components may beinserted to those noted above. Depending upon the embodiment, thearrangement of components may be interchanged with others replaced.

In some examples, the lesion identification module 603 and/or thegrading network module 604 can be implemented in a variety of way, suchas be implemented in software or firmware, including software code beingstored in memory 602 and executable by processor 601, or in one or morehardware devices including general purpose integrated circuits,dedicated circuits, field programmable gate arrays, and the like. Insome examples, the above components are coupled and communicated witheach other by, for example, a bus or other mechanism.

In various examples, the input/output interface 605 is configured toreceive X-ray chest radiographs and/or to transmit (or pass) data to thelesion identification module 603 and/or the grading module 604 foroutputting the classification result. In some embodiments, the lesionidentification module 603 is configured to be trained (e.g., machinelearning) using training data, and after training, to be used to outputa list of lesion candidates and/or corresponding score(s) based on theinput chest radiographs. In certain embodiments, the grading module 604is configured to use the training data for machine learning and, aftertraining, be used to output one or more grading results based on thelist of lesion candidates and/or their corresponding score(s) generatedby the lesion identification module 603. In some examples, the lesionidentification module 603 is a separate (e.g., standalone) module and isnot part of the grading system 600.

FIG. 7 is an illustrative representation of an interface of a system forgrading an X-ray chest radiograph by severity and/or urgency, accordingto some embodiments of the present invention. As shown, the interfaceincludes an AI-TRIAGE column, where a circular lung icon with anexclamation point (shown in the fourth cell down) represents a chestradiograph require attention (e.g., a problematic chest radiograph),indicating its higher level of concern when compared to the rest of thechest radiographs. In some examples, the icon is prompted forpreferential reading and diagnosis for chest radiographs assigned with“critical”, “more urgent”, or “require attention”, such as in method 400or in method 500. In certain examples, chest radiographs without theicon are ones assigned with a grade of “no notable abnormality” and areprompted for time-divisional and/or labor-divisional reading anddiagnosis.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or”. That is, the phrase “X” employs “A” or “B” tomean any natural collocation, unless otherwise indicated. That is, thephrase “X” using “A” or “B” is satisfied by any of the followingexamples: X employs A; X employs B; or X employs both A and B. By“connected” and “coupled” are meant to mean the same, meaning theelectrical connection of the two devices. In addition, the articles “a”,“an” and “the” can be nonlimiting and interpreted as one or more.

Various aspects or features will be presented in the form of a systemthat can include several devices, components, modules, and the like. Itis to be understood and appreciated that the various systems may includeadditional devices, components, modules, etc. and/or may not include allof the devices, components, modules, etc. discussed in connection withthe Figures. A combination of these methods can also be used.

Various illustrative logic, logic blocks, modules, and circuitsdescribed in connection with the embodiments disclosed herein may beused in general purpose processors, digital signal processors (DSPs),application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs). Or other programmable logic devices, discrete gateor transistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A generalpurpose processor may be a microprocessor, but in the alternative, theprocessor may be any conventional processor, controller,microcontroller, or state machine. The processor may also be implementedas a combination of computing devices, such as a combination of a DSPand a microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration. Furthermore, at least one processor can include one ormore modules operable to perform one or more of the steps and/or actionsdescribed above. For example, the embodiments described above inconnection with various methods can be implemented by a processor and amemory coupled to the processor, where the processor can be configuredto perform any of the steps of any of the methods described above, orany combination thereof.

Furthermore, the steps and/or actions of a method or algorithm describedin connection with the aspects disclosed herein can be implementeddirectly in hardware, in a software module executed by a processor, orin a combination of the two. For example, the embodiments describedabove in connection with the various methods can be implemented by acomputer readable medium storing computer program code, which whenexecuted by a processor/computer performs any of the steps of any of themethods described above, or any combination thereof.

In various embodiments, a system for grading a medical image includes agrading network configured to provide a grading result corresponding tothe medical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network. In someexamples, the system is implemented according to at least the gradingsystem 600 of FIG. 6 and/or configured to perform at least the method100 of FIG. 1, the method 300 of FIG. 3A, the method 400 of FIG. 4,and/or the method 500 of FIG. 5.

In some embodiments, the grading network is further configured toprovide the grading result corresponding to the medical image based onat least the medical image and the list of lesion candidates generatedby the lesion identification network.

In some embodiments, the grading network is further configured toprovide the grading result corresponding to the medical image based onat least the medical image.

In some embodiments, the lesion identification network is configured toprovide a score corresponding to the list of lesion candidates. In someexamples, the grading result is provided further based on the score. Incertain examples, the score is provided based on at least a probabilityvalue corresponding to the medical image including lesions of multiplelesion categories.

In some embodiments, the medical image includes a chest radiograph. Insome examples, the list of lesion candidates includes at least one listselected from a group consisting of a list of lung lesion candidates, alist of cardiac lesion candidates, a list of cardiovascular lesioncandidates, a list of hilar lesion candidates, a list of bone tissuelesion candidates, and a list of pleural lesion candidates.

In some embodiments, the grading network is further configured toprovide one or more outputs to guide manual scheduling of a plurality ofradiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs. In some examples, the grading network is further configuredto automatically schedule the plurality of radiographs for manual reviewand diagnosis based on at least a plurality of grading resultscorresponding to the plurality of radiographs.

In some embodiments, the grading network is further configured toprovide one or more outputs to guide manual scheduling of a plurality ofradiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs, and automatically schedule the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.

In some embodiments, the grading network is a student network trained byan attention transfer learning process includes: establishing a teachernetwork and the student network for the grading network; training theteacher network; and training the student network based on at least:extracting a feature map from one or more middle layers corresponding toboth the student network and the teacher network; calculating one ormore attention transfer learning losses corresponding to the one or moremiddle layers; and backpropagating the one or more attention transferlearning losses into the student network.

In some embodiments, the grading result corresponds to at least oneselected from a group consisting of severity and urgency.

In some embodiments, the lesion identification network is configured toidentify at least one lesion characteristic selected from a groupconsisting of color, shape, size, grayscale value, position, andmorphology.

In some embodiments, the grading result is provided based on the medicalimage as a whole or one or more partial regions of the medical image.

In some embodiments, the grading result is selected from a groupconsisting of a first grade, a second grade, and a third grade, wherein:the first grade corresponds to a first priority for reading anddiagnosis; the second grade corresponds to a second priority for readingand diagnosis; the third grade corresponds to a third priority forreading and diagnosis; the first priority being greater than the secondand third priorities.

In various embodiments, a computer-implemented method for grading amedical image includes providing a grading result corresponding to themedical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network. In someexamples, the method is implemented according to at least the method 100of FIG. 1, the method 300 of FIG. 3A, the method 400 of FIG. 4, and/orthe method 500 of FIG. 5. In certain examples, the method is implementedby at least the grading system 600 of FIG. 6.

In some embodiments, the computer-implemented method further includesproviding the grading result corresponding to the medical image based onat least the medical image and the list of lesion candidates generatedby the lesion identification network.

In some embodiments, the computer-implemented method further includesproviding the grading result corresponding to the medical image based onat least the medical image.

In some embodiments, the computer-implemented method further includesproviding a score corresponding to the list of lesion candidates usingthe lesion identification network. In some examples, the providing agrading result is further based on the score. In certain examples, theproviding a score is based on at least a probability value correspondingto the medical image including lesions of multiple lesion categories.

In some embodiments, the computer-implemented method further includesproviding one or more outputs to guide manual scheduling of a pluralityof radiographs for manual review and diagnosis based on at least aplurality of grading results corresponding to the plurality ofradiographs. In some examples, the computer-implemented method furtherincludes automatically scheduling the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.

In some embodiments, the computer-implemented method further includestraining a student network to be the grading network based on anattention transfer learning process including: establishing a teachernetwork and the student network for the grading network; training theteacher network; and training the student network based on at least:extracting a feature map from one or more middle layers corresponding toboth the student network and the teacher network; calculating one ormore attention transfer learning losses corresponding to the one or moremiddle layers; and backpropagating the one or more attention transferlearning losses into the student network.

In some embodiments, scheduling of the plurality of radiographs formanual review and diagnosis is based on at least one selected from agroup consisting of expertise, pay level, and seniority.

In various embodiments, a non-transitory computer-readable medium withinstructions stored thereon, that when executed by a processor, performthe process of: providing a grading result corresponding to the medicalimage based on at least the medical image and/or a list of lesioncandidates generated by a lesion identification network. In someexamples, the non-transitory computer-readable medium with instructionsstored thereon is implemented according to at least the method 100 ofFIG. 1, and/or a computer (e.g., a terminal).

For example, some or all components of various embodiments of thepresent invention each are, individually and/or in combination with atleast another component, implemented using one or more softwarecomponents, one or more hardware components, and/or one or morecombinations of software and hardware components. In another example,some or all components of various embodiments of the present inventioneach are, individually and/or in combination with at least anothercomponent, implemented in one or more circuits, such as one or moreanalog circuits and/or one or more digital circuits. In yet anotherexample, while the embodiments described above refer to particularfeatures, the scope of the present invention also includes embodimentshaving different combinations of features and embodiments that do notinclude all of the described features. In yet another example, variousembodiments and/or examples of the present invention can be combined.

Additionally, the methods and systems described herein may beimplemented on many different types of processing devices by programcode including program instructions that are executable by the deviceprocessing subsystem. The software program instructions may includesource code, object code, machine code, or any other stored data that isoperable to cause a processing system to perform the methods andoperations described herein. Other implementations may also be used,however, such as firmware or even appropriately designed hardwareconfigured to perform the methods and systems described herein.

The systems' and methods' data (e.g., associations, mappings, datainput, data output, intermediate data results, final data results, etc.)may be stored and implemented in one or more different types ofcomputer-implemented data stores, such as different types of storagedevices and programming constructs (e.g., RAM, ROM, EEPROM, Flashmemory, flat files, databases, programming data structures, programmingvariables, IF-THEN (or similar type) statement constructs, applicationprogramming interface, etc.). It is noted that data structures describeformats for use in organizing and storing data in databases, programs,memory, or other computer-readable media for use by a computer program.

The systems and methods may be provided on many different types ofcomputer-readable media including computer storage mechanisms (e.g.,CD-ROM, diskette, RAM, flash memory, computer's hard drive, DVD, etc.)that contain instructions (e.g., software) for use in execution by aprocessor to perform the methods' operations and implement the systemsdescribed herein. The computer components, software modules, functions,data stores and data structures described herein may be connecteddirectly or indirectly to each other in order to allow the flow of dataneeded for their operations. It is also noted that a module or processorincludes a unit of code that performs a software operation and can beimplemented for example as a subroutine unit of code, or as a softwarefunction unit of code, or as an object (as in an object-orientedparadigm), or as an applet, or in a computer script language, or asanother type of computer code. The software components and/orfunctionality may be located on a single computer or distributed acrossmultiple computers depending upon the situation at hand.

The computing system can include client devices and servers. A clientdevice and server are generally remote from each other and typicallyinteract through a communication network. The relationship of clientdevice and server arises by virtue of computer programs running on therespective computers and having a client device-server relationship toeach other.

This specification contains many specifics for particular embodiments.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations, one or more features from a combination can in some casesbe removed from the combination, and a combination may, for example, bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Although specific embodiments of the present invention have beendescribed, it will be understood by those of skill in the art that thereare other embodiments that are equivalent to the described embodiments.Accordingly, it is to be understood that the invention is not to belimited by the specific illustrated embodiments.

What is claimed is:
 1. A system for grading a medical image, the systemcomprising: a grading network configured to provide a grading resultcorresponding to the medical image based on at least the medical imageand/or a list of lesion candidates generated by a lesion identificationnetwork; wherein the grading network is further configured to at leastone of: provide one or more outputs to guide manual scheduling of aplurality of radiographs for manual review and diagnosis based on atleast a plurality of grading results corresponding to the plurality ofradiographs; or automatically schedule the plurality of radiographs formanual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.
 2. The system ofclaim 1, wherein: the grading network is further configured to providethe grading result corresponding to the medical image based on at leastthe medical image and the list of lesion candidates generated by thelesion identification network.
 3. The system of claim 1, wherein: thegrading network is further configured to provide the grading resultcorresponding to the medical image based on at least the medical image.4. The system of claim 1, wherein: the lesion identification network isconfigured to provide a score corresponding to the list of lesioncandidates; the grading result is provided further based on the score;and the score is provided based on at least a probability valuecorresponding to the medical image including lesions of multiple lesioncategories.
 5. The system of claim 1, wherein: the medical imageincludes a chest radiograph; and the list of lesion candidates includesat least one list selected from a group consisting of a list of lunglesion candidates, a list of cardiac lesion candidates, a list ofcardiovascular lesion candidates, a list of hilar lesion candidates, alist of bone tissue lesion candidates, and a list of pleural lesioncandidates.
 6. The system of claim 1, wherein the grading network is astudent network trained by an attention transfer learning processcomprising: establishing a teacher network and the student network forthe grading network; training the teacher network; and training thestudent network based on at least: extracting a feature map from one ormore middle layers corresponding to both the student network and theteacher network; calculating one or more attention transfer learninglosses corresponding to the one or more middle layers; andbackpropagating the one or more attention transfer learning losses intothe student network.
 7. The system of claim 1, wherein the gradingresult corresponds to at least one selected from a group consisting ofseverity and urgency.
 8. The system of claim 1, wherein the lesionidentification network is configured to identify at least one lesioncharacteristic selected from a group consisting of color, shape, size,grayscale value, position, and morphology.
 9. The system of claim 1,wherein the grading result is provided based on the medical image as awhole or one or more partial regions of the medical image.
 10. Thesystem of claim 1, wherein the grading result is selected from a groupconsisting of a first grade, a second grade, and a third grade, wherein:the first grade corresponds to a first priority for reading anddiagnosis; the second grade corresponds to a second priority for readingand diagnosis; the third grade corresponds to a third priority forreading and diagnosis; the first priority being greater than the secondand third priorities.
 11. A computer-implemented method for grading amedical image, the method comprising: providing a grading resultcorresponding to the medical image based on at least the medical imageand/or a list of lesion candidates generated by a lesion identificationnetwork; and at least one of: providing one or more outputs to guidemanual scheduling of a plurality of radiographs for manual review anddiagnosis based on at least a plurality of grading results correspondingto the plurality of radiographs; or automatically scheduling theplurality of radiographs for manual review and diagnosis based on atleast a plurality of grading results corresponding to the plurality ofradiographs.
 12. The computer-implemented method of claim 11, furtherincluding: providing the grading result corresponding to the medicalimage based on at least the medical image and the list of lesioncandidates generated by the lesion identification network.
 13. Thecomputer-implemented method of claim 11, further including: providingthe grading result corresponding to the medical image based on at leastthe medical image.
 14. The computer-implemented method of claim 11,further includes: providing a score corresponding to the list of lesioncandidates using the lesion identification network; wherein theproviding a grading result is further based on the score; and whereinthe providing a score is based on at least a probability valuecorresponding to the medical image including lesions of multiple lesioncategories.
 15. The computer-implemented method of claim 11, furtherincludes training a student network to be the grading network based onan attention transfer learning process comprising: establishing ateacher network and the student network for the grading network;training the teacher network; and training the student network based onat least: extracting a feature map from one or more middle layerscorresponding to both the student network and the teacher network;calculating one or more attention transfer learning losses correspondingto the one or more middle layers; and backpropagating the one or moreattention transfer learning losses into the student network.
 16. Thecomputer-implemented method of claim 11, wherein the scheduling of theplurality of radiographs for manual review and diagnosis is based on atleast one selected from a group consisting of expertise, pay level, andseniority.
 17. A non-transitory computer-readable medium withinstructions stored thereon, that when executed by a processor, performthe processes of: providing a grading result corresponding to themedical image based on at least the medical image and/or a list oflesion candidates generated by a lesion identification network; and atleast one of: providing one or more outputs to guide manual schedulingof a plurality of radiographs for manual review and diagnosis based onat least a plurality of grading results corresponding to the pluralityof radiographs; or automatically scheduling the plurality of radiographsfor manual review and diagnosis based on at least a plurality of gradingresults corresponding to the plurality of radiographs.